JAMIA Open

Background Machine learning (ML) models are increasingly used to predict adverse outcomes after surgery. However, most rely on static patient characteristics (e.g., age, comorbidities) and overlook clinician-controlled treatment decisions that can be actively modified at the point of care. Discharge opioid prescribing is a key modifiable, clinician-controlled decision, yet optimizing prescribing choices across multiple adverse outcomes remains underexplored in predictive modeling. This study addresses that gap by introducing a novel ML framework that explicitly separates fixed patient risk factors from modifiable prescribing options to support personalized, risk-informed opioid prescribing decisions. Methods We developed the Hierarchical Clinical Fusion Transformer (HCF-Transformer), an ML model designed to estimate patient-specific risks across four postoperative outcomes: prolonged opioid use (POU), chronic pain (CP), 30-day readmission, and opioid-associated outcomes (OAO). The model constructs patient risk profiles from fixed, non-modifiable baseline factors, followed by a transformer layer. Clinician-controllable discharge opioid regimens are modeled as alternative intervention candidates and fused with the fixed risk representation through a clinical fusion mechanism, enabling assessment and ranking based on predicted risks. A Total Relative Risk (TRR) metric, calibrated to each outcome prediction threshold, guides the recommendation process. We evaluated the model in diabetic surgical patients, a common high-risk population. Results The study included 157,853 unique diabetic surgical patients, with outcome prevalences ranging from 47.2% (POU) to 1.8% (OAO). The HCF-Transformer achieved the highest AUROCs, 0.798 for POU, 0.712 for 30-day readmission, 0.808 for CP, and 0.922 for OAO, outperforming Random Forest, FT-Transformer, and ResNet-based models. Compared to these baselines, HCF-Transformer generated more stable and discriminative risk estimates and demonstrated significant variation in TRR scores across discharge opioid options (ANOVA p < .01, eta-squared > .01). This enabled consistent identification of lower-risk regimens tailored to patient-specific profiles. Conclusions The HCF-Transformer introduces a novel hierarchical fusion approach to optimize opioid prescribing by integrating static patient risk profiles with modifiable discharge options. Using transformer-based modeling and a quantifiable TRR metric, the model delivers personalized, risk-aware recommendations. This approach enables data-driven opioid prescribing tailored to individual risk and has the potential to improve postoperative outcomes in high-risk populations. Our findings demonstrate that integrating modifiable factors with structured risk profiles through a transformer-based fusion architecture can enhance decision-support systems, paving the way for more actionable and personalized AI in healthcare.

We present a data-driven framework to predict 15-year all-cause mortality using outpatient administrative records for 2.3 million Veterans in the largest integrated U.S. healthcare system. Rather than relying on predefined clinical phenotypes, we used the 1,000 most common outpatient medical codes from each of three data types/modalities: ICD-9 (Dx), Current Procedural Terminology (CPT), and prescription drugs (Rx), encoded as binary features. Using these features, we trained three machine learning (ML) algorithms (logistic regression with lasso, random forest, and a 3-layered feed-forward neural network) to predict 15-year mortality risk. The features were also mapped to variables for the widely used Charlson Comorbidity Index (CCI), Elixhauser, and Veterans Aging Cohort Study (VACS) indices, refitted for 15-year mortality prediction, for baseline comparison. All our models significantly outperformed the widely used CCI, Elixhauser, and VACS indices, with C-statistics ranging from 0.82 to 0.84 versus 0.739 to 0.804 for the baselines. Relative improvements in C-statistics of our approach over the baselines were consistent across different subgroups (age groups of <65 years, those 65+years, Blacks, Hispanics, etc.) Our approach enabled the identification of high-impact predictors with clinical grounding , without requiring hand-curated phenotypes. Cardiovascular diseases and mental health diagnoses/treatments emerged as leading long-term mortality indicators. Using unsupervised ML techniques including PCA and K-means clustering, we associated interpretable patterns and complex interactions between diagnoses and treatments, highlighting comorbidities, disease trajectories, and healthcare utilization patterns. The ability to achieve the predictive performance and algorithmically detect such relationships purely from outpatient data supports the scalability and broad applicability of our framework. This framework not only improves mortality risk stratification over existing clinical indices, but also enables better understanding of how medical codes, regardless of category, interact to predict long-term outcomes.

Objective: Clinical phenotyping methods that rely on clinical and informatics expertise can be time-intensive and costly. We tested both manual and highly automated approaches using electronic health record (EHR) data to identify an FDA Sentinel Initiative health outcome of interest, acute pancreatitis. Materials and Methods: We trained and evaluated machine learning algorithms using EHR data with two approaches: a custom approach that included manually curated features and trained on outcomes data validated with medical record review, and a highly automated approach that greatly simplifies and automates feature engineering and relies on low-cost silver-standard outcomes for model training. Results: Custom algorithms using manually curated structured claims data discriminated cases from non-cases with a high degree of accuracy (cv-AUC 0.89 [95%CI 0.84-0.94]); the inclusion of natural language processing (NLP)-derived covariates from clinical notes increased performance slightly (cv-AUC 0.91[95%CI 0.86-0.97]). The automated algorithm trained on the outcome count of diagnosis codes performed less well (AUC 0.80 [95% CI 0.75-0.85]) but improved using maximum lipase value as an outcome (AUC 0.88 [95% CI 0.84-0.92]). At a positive predictive value of 90%, the custom algorithm had a sensitivity of 92%, the automated algorithm trained on diagnosis code count had a sensitivity of 45%, and the automated algorithm trained on maximum lipase value had a sensitivity of 84%. However, a prediction rule derived by clinicians during chart review was nearly as accurate (maximum lipase value [&ge;] 3 times upper limit of normal; AUC 0.86, PPV 85%, sensitivity 92%). Discussion: Machine learning algorithms with manually curated structured data and NLP features trained on validated outcomes data successfully identified validated events. Use of an outcome in the automated model based on specific phenotype knowledge (maximum lipase value) allowed for performance similar to the custom model and with considerably less resources.

Cancer registries enable cancer surveillance at the population level. These registries require significant human-time to read through many different parts of the electronic health record, including structured data and lengthy, free-text clinical reports, to abstract values for hundreds of required variables. Large language models (LLMs) offer the possibility to significantly improve this process by supporting and speeding up cancer registry data abstraction. However, it is unclear how well these models perform at real-world cancer registry abstraction involving multiple cancer types and large patient volumes. Here, we evaluate five foundational LLMs for their ability to reliably abstract cancer registry variables. We leverage hospital cancer registry data from a large regional health system as the ground truth and use LLMs to abstract from clinical reports eight registry variables for 5,939 patients with seven different cancer types. We use a zero-shot prompting strategy to compare LLM ability on commonly abstracted cancer variables with different data types. The results show that larger and more advanced models (Claude Sonnet 4.5, GPT-OSS-120b, GPT-OSS-20b) generally outperform smaller models (Gemma 12b, LLaMA 3.1 8b). The best performing models show F1 scores around 0.8 for cancer registry variables with low cardinality (grade, summary stage, laterality), with only slightly lower F1 scores for variables with high cardinality (primary site, regional nodes examined, regional nodes positive). On the more complex task of precise date extraction, all models showed decreased performance on both diagnosis and treatment dates (exact accuracy ~0.55 for the best performing models), which increased to ~0.85 for a tolerance within {+/-}30 days. These results quantify the performance of various models as well as the potential and limitations of LLMs in cancer registry abstraction tasks.

Background: Acute kidney injury (AKI) is a severe complication in intensive care units, frequently exacerbated by synergistic nephrotoxicity from drugs such as Vancomycin and Piperacillin-Tazobactam. Traditional alert systems relying on static thresholds suffer from high false-positive rates and delayed detection. Methods: We developed Agentic-TimesFM-AKI, a dual-model architecture integrating a Large Language Model (Gemma-4 Sentinel) with a zero-shot time-series forecaster (TimesFM) to provide continuous, dynamic risk forecasting and transparent clinical reasoning. The system was trained on a synthetically generated cohort with differential privacy ({varepsilon}=10) and evaluated on the publicly accessible eICU (N=200) and MIMIC-IV (N=117) Demo datasets. Results: In the internal eICU pilot evaluation, the framework achieved an Accuracy of 0.970 (95% CI: 0.945-0.990) and an F1-Score of 0.966, successfully mapping temporal physiological trajectories into intelligible natural language alerts. However, external validation on the MIMIC-IV cohort revealed severe performance degradation. Conclusions: While the dual-model framework provides highly accurate and interpretable AKI alerts on familiar schema cohorts, it suffers from structural formatting fragility and domain shift. This highlights critical vulnerabilities in applying generative models to out-of-distribution electronic health records. Keywords: Acute Kidney Injury, Large Language Models, Time-Series Forecasting, Electronic Health Records, Differential Privacy, Pharmacovigilance.

Background Peripheral artery disease (PAD) is a major cause of cardiovascular events but remains underdiagnosed. Electronic health record (EHR)-based machine learning models show promise for earlier detection, but developing generalizable and fair models across diverse populations remains challenging. Methods Using the University of California Health Data Warehouse, containing EHR data from five health systems, we identified patients with and without PAD. We used unsupervised clustering to define PAD phenotypes and trained a LightGBM classifier using 14,023 features spanning demographics, comorbidities, medications, laboratory values, healthcare utilization, and diagnosis, procedure, and medication codes. We evaluated performance overall and across demographic groups and phenotypes, and assessed fairness using selection rates and subgroup differences in true- and false-positive rates. Results The study included 33,739 cases and 33,739 matched controls. Clustering identified four phenotypes: patients with limited healthcare documentation (cluster 1), younger patients with severe metabolic disease (cluster 2), patients with a traditional atherosclerotic risk profile (cluster 3), and frail elderly patients with multimorbidity (cluster 4). Overall, the model demonstrated consistent performance across institutions (AUROC 0.76?0.79; AUC-PR 0.76?0.79) with well-calibrated probabilities. Performance was similar across genders, with modest variation by race and age, and was stronger in clusters 2?4. Cluster 2 demonstrated the highest sensitivity (TPR 0.87, 95% CI 0.87?0.88), while cluster 1 showed the lowest performance (TPR 0.40, 95% CI 0.39?0.41). Conclusions The EHR-based PAD detection model demonstrated consistent performance across five health systems. Phenotypic clustering revealed clinically meaningful differences in model performance adding an additional consideration in ML fairness and performance evaluations.

Background: Heart failure (HF) and chronic obstructive pulmonary disease (COPD) are among the leading causes of morbidity and mortality globally, with effective management heavily dependent on accurate severity staging using the New York Heart Association (NYHA) and Global Initiative for Chronic Obstructive Lung Disease (GOLD) classification systems. However, severity information is frequently embedded within unstructured clinical narratives rather than standardized Electronic Health Record (EHR) fields, limiting automated clinical decision support, disease surveillance, and retrospective healthcare analytics. Existing Natural Language Processing (NLP) approaches primarily rely on rule-based keyword extraction or supervised deep learning methods requiring large annotated corpora, which are often unavailable in many healthcare settings. Equally, most current systems inadequately integrate clinical ontologies for semantic reasoning and explainable classification, limiting interoperability and clinical applicability. Objective: This study aims to develop and evaluate an ontology-integrated NLP framework for automated extraction and severity staging of HF and COPD symptoms from de-identified clinical notes using NYHA and GOLD classification systems. Methods: The study will employ a Design Science Research (DSR) methodology to design, implement, and evaluate a hybrid NLP framework integrating rule-based extraction, SNOMED-CT ontology reasoning, and a Bidirectional Long Short-Term Memory with Conditional Random Field (Bi-LSTM-CRF) deep learning architecture for clinical sequence labeling. Approximately 1,000 de-identified clinical notes will be sampled proportionately from publicly available repositories including MIMIC-III/IV, eICU Collaborative Research Database, AmsterdamUMCdb, and MTSamples. Clinical text preprocessing will include tokenization, lemmatization, dependency parsing, abbreviation expansion, and negation detection. Ontology-guided semantic normalization will map extracted symptom entities to standardized SNOMED-CT concepts to support severity staging. Framework performance will be evaluated using precision, recall, F1-score, Cohens Kappa, sensitivity, specificity, positive predictive value, negative predictive value, confusion matrices, and correlation analyses against confirmed diagnoses and guideline-based severity classifications. Expected Outcomes: The proposed framework is expected to automate NYHA and GOLD severity staging across heterogeneous clinical note types without reliance on manually annotated severity labels. The ontology-integrated architecture is anticipated to improve semantic consistency, interpretability, and explainability of NLP outputs while enhancing EHR analytics, retrospective clinical audit, and AI-assisted clinical decision support. Conclusion: Findings from this study may provide a scalable and transferable framework for automated severity classification in data-rich but label-poor healthcare environments.

Objectives Transformer models for electronic health records require converting clinical data into token sequences, however standardized tokenization and evaluation frameworks are lacking. We introduce DBToken, an open-source library, and bits-per-row (BPR), a metric for comparing tokenization strategies. Materials and Methods DBToken accepts Medical Event Data Standard (MEDS)-compatible input and supports multiple text, numeric, and temporal tokenization strategies. BPR extends the bits-per-byte metric used in language models to enable comparison across tokenization strategies. Results DBToken efficiently tokenized data across configurations. BPR identified the vocabulary size associated with the best clinical outcome performance and localized differences in numeric tokenization performance by token class. Discussion Optimal tokenization strategies for medical foundation models are a subject of active research. DBToken enables reproducible tokenization experiments, while BPR efficiently screens vocabulary sizes and numeric representations before downstream evaluation. Conclusion DBToken and the BPR metric provide open-source infrastructure for reproducible EHR tokenization and cross-strategy evaluation.

Background: Unstructured data represent about 80% of total electronic health records (EHR) data. Structuring this free text is essential for advancing clinical research, including cohort selection for trials, retrospective studies, and the development of disease registries. While manual chart review (MCR) remains the gold standard for extracting this clinical data, the process is inherently slow, resource-intensive, and susceptible to errors from human fatigue. We evaluated the extraction accuracy, safety, and efficiency of the HeLIX (Hepatology Logic-Integrated Extraction) framework, a Large Language Model (LLM) protocol using Google Gemini 3 Pro, compared to a gold-standard Manual Chart Review (MCR). Methods: A prospective validation study was conducted using 50 high-complexity, simulated hepatology discharge summaries designed to replicate the real-world heterogeneity of EHRs. The HeLIX framework employed a Zero-Shot, Structured Chain-of-Thought (CoT) prompting strategy enforced by a three-layer architecture: Clinical Reasoning Trace, Schema Enforcement, and Evidence Verification. The model extracted 45 distinct clinical variables. Performance was benchmarked against a consensus MCR. Results: Across 2,250 evaluated data points, the model achieved an overall Extraction Accuracy of 99.24% (95% CI: 98.8%-99.5%), with perfect concordance in 35/45 (77.8%) variables. For binary diagnostic variables, the model demonstrated an overall F1-score of 0.98, Recall of 0.99 and substantial inter-rater reliability (Cohens {kappa} = 0.97). Hallucinations were exceptionally rare (2/2250; 0.08%). Critical errors affecting clinical management occurred in only 2 instances (<0.1% of total data), both involving etiological misattribution in complex multifactorial diagnoses. The AI workflow was 13.4-fold faster and 95.1% more cost-effective than manual extraction. Conclusion: The HeLIX framework demonstrates physician-level accuracy and reliability in extracting complex hepatology data. It offers a scalable, efficient, and economical alternative to manual chart review. Such frameworks could accelerate clinical research, enabling healthcare systems globally to build comprehensive patient registries for a fraction of the traditional cost.

The COVID-19 pandemic highlighted the need for interoperable health data infrastructures supporting reproducible observational research. The Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) provides a widely adopted standard for harmonizing heterogeneous health data, but adoption remains limited in francophone Africa where language barriers and non-standardized surveillance systems pose additional challenges. We developed a complete Extract-Transform-Load (ETL) pipeline to convert a heterogeneous Senegalese COVID-19 surveillance dataset into OMOP CDM version 5.4. Source data recorded in French were translated into English through an iterative process interleaved with vocabulary mapping using ATHENA and Usagi. Semantic standardization used SNOMED CT for conditions, LOINC for measurements, and RxNorm for drugs. All 214 mappings underwent expert review by clinical and data science specialists. Data quality was assessed using the OHDSI Data Quality Dashboard (DQD) and Achilles. The standardized database achieved complete transformation (100%) for eight of the eleven source-populated domain tables, including person, visit_occurrence, measurement, and death. Partial transformation was observed for condition_occurrence (95.3%) and observation (68.1%), primarily due to incomplete vocabulary coverage for occupation categories and context-specific variables. The DQD produced an overall pass rate of 97% and a corrected pass rate of 98%, comparable to other published African OMOP implementations. Among the 19 data-quality failures, conformance and completeness issues predominated; the conformance failures were largely foreign-key checks, reflecting placeholder concept values (concept_id = 0) for metadata fields without meaningful equivalents in surveillance data. Iterative translation refinement was required when French-to-English translations did not align with OHDSI vocabulary terminology. This work documents, to our knowledge, the first OMOP CDM implementation on COVID-19 surveillance data in Senegal and francophone West Africa and provides a reusable methodological blueprint for future OMOP deployments in the region.

Background: Increased public access to data from disparate sources provides opportunities to study and validate predictive and subphenotype models in heterogeneous disease conditions using aggregated individual patient data. Robust, explicit, and transparent harmonization of data elements is critical to ensure interpretability, reproducibility, and generalizability of secondary and retrospective analyses. Methods & Results: We designed and implemented ADAPT (Aggregating Data to Accelerate Personalized Therapy), a scalable framework using multiple software packages (R, SQL, BigQuery) that enables rapid, explicit harmonization of structured data elements from randomized trials and observational studies using a standard spreadsheet interface. User-specified criteria are applied to primary study data to produce harmonized longitudinal datasets comprised of demographics, medical history, quantitative observations, repeated measures, and clinical outcomes. We demonstrate this functionality using 26 clinical studies found in the National Heart, Lung, and Blood Institute BioLINCC resource. We illustrate the scalability of ADAPT to the order of billions of datapoints using administrative clinical data in a cloud-computing platform. We also present examples of collaborators using ADAPT for independent harmonization tasks for secondary analyses and democratization of publicly available data. Conclusion: ADAPT is a disease-agnostic, extensible, and scalable platform to support robust, transparent harmonization of structured research data using interfaces accessible to a variety of researchers regardless of programming ability. It extends FAIR principles beyond research data to also represent harmonization analyses by improving Findability of harmonization decisions, Accessibility of methods to other stakeholders, Interoperability with independent analyses and datasets, and Reusability through efficient implementation in a variety of analysis environments.

Stigmatizing language in clinical documentation, which conveys negative stereotypes, attitudes, or judgments toward patients, is a recognized source of documentation bias and is associated with poorer care and adverse health outcomes. Although prior stigma-related research has focused on clinician-written EHR notes, the increasing use of large language model (LLM)-generated documentation in clinical workflows raises new concerns about its potential to reproduce or amplify bias and affect patient safety. In this study, we conducted a large-scale assessment of stigmatizing language in LLM-generated reasoning text on 35 real-world clinical tasks across 107 LLMs. We applied a psychiatrist-validated, natural language processing (NLP) system to detect stigma terms in LLM reasoning text and quantified stigma rates of LLM-generated reasoning texts across 3,745 model-task pairs. Results showed that stigma rates ranged from 0% to 33.33%, with 84.06% of pairs containing stigma terms. Open-source models and reasoning models showed statistically higher stigma rates than proprietary (1.97% vs. 1.60%; p < 0.01) and non-reasoning models (2.35% vs. 1.70%; p < 0.0001), while the stigma rate difference between the general and medical models is not statistically significant (2.00% vs. 1.80%; p = 0.26). Stigma rates of LLM outputs correlated negatively with task accuracy (r = -0.304; p < 0.001) and positively with input clinical-text stigma (r = 0.569; p < 0.001), with 19.76% of model-task pairs amplifying stigma in the original input notes. Applying prompt engineering as a destigmatizing approach helped reduce model stigma rates by as much as 91.91% without affecting the model performance. This study shows that stigmatizing language generation is common but reducible during LLMs' reasoning traces, suggesting that well-implemented approaches for LLM monitoring and destigmatizing will be essential for healthcare systems to implement.

We present FHIRBench, a benchmark evaluating six FHIR clinical data serialization strategies across four frontier LLMs (Claude Sonnet 4.5, GPT-5.4, DeepSeek V3.2, Qwen3 32B) on three clinical tasks using 100 stratified synthetic FHIR R4 patient bundles. We employ two evaluation layers: token-level F1 and LLM-as-judge rubric on four clinical dimensions, yielding 7,200 evaluations per layer. Our findings reveal four results. First, serialization significantly impacts quality but the direction diverges between layers: Condensed outperforms Raw JSON on F1 for 3/4 models (Wilcoxon p < 10^-17), while Raw JSON achieves higher judge scores for 3/4 models (p < 10^-7). Narrative achieves 95% of Raw JSON's quality at 83% fewer tokens. Second, model rankings completely reverse between layers -- Claude ranks last on F1 but first on clinical quality (p = 1.0 x 10^-6), demonstrating that single-metric evaluation produces misleading model selection. Third, a significant Model x Serializer interaction (Friedman p = 0.0009) precludes universal format recommendations, with GPT-5.4 favoring Raw JSON while open-weight models favor compressed formats. Fourth, Llama 3.1 70B exhibits 100% inference failure on complex patients despite operating within its nominal context window, revealing a patient-safety gap where AI fails for the patients who need it most. These findings establish that clinical AI systems require model-aware serialization middleware, multi-layer evaluation frameworks, and capacity verification before deployment. Code and data publicly available.

Existing evaluations of healthcare AI often treat interoperability as a technical infrastructure issue rather than a factor that directly influences the safety and reliability of clinical AI systems. Yet the quality of Fast Healthcare Interoperability Resources (FHIR) implementation affects whether AI models can operate accurately, fairly, securely, and effectively in real clinical settings. We present FHIRTrustBench, a benchmark for assessing the readiness of FHIR-based clinical AI systems across five complementary dimensions: FHIR implementation quality, AI validation, clinical workflow integration, trustworthiness assessment, and governance readiness. Each dimension is mapped to a distinct category of downstream deployment failure risk. We applied FHIRTrustBench to a corpus of 10 representative sources spanning interoperability standards, implementation studies, electronic health record integration research, healthcare large language model research, and governance frameworks. Each source was scored individually and traceably against the five-dimension rubric. FHIR Specificity achieved the highest dimension mean at 1.3 out of 2.0, while AI Validation received the lowest at 0.3. Even category-leading sources that scored a maximum 2.0 on FHIR Specificity scored 0 on AI Validation. Prospective external validation was reported in no source, and Governance Readiness remained at or below 1.0 across every category. We further identify five interoperability-related AI failure pathways, spanning data integrity, semantic consistency, security, clinical workflow, and generative AI grounding, and propose a deployment lifecycle framework and reporting checklist that translate benchmark scores into deployment-readiness decisions for developers, healthcare organizations, and regulators. FHIRTrustBench provides a practical and reproducible basis for assessing FHIR-enabled clinical AI before deployment and can evolve as interoperability standards and clinical evidence mature.

Health-related social needs (HRSNs), such as housing instability, food insecurity, and transportation challenges, are nonmedical factors associated with poorer health and well-being. Screening for unmet HRSNs is a critical step towards identifying at-risk patients, but manual screening is resource intensive and often incomplete. We utilized Electronic Health Records (EHR) data to develop machine learning models to identify unmet HRSNs using a limited set of non-modifiable sociodemographic features available in EHRs. We included 745,975 patients screened for at least one HRSN using data from community health centers that participated in the OCHIN practice-based research network between 2016 and 2022. Logistic regression, random forest (RF), eXtreme Gradient Boosting (XGBoost), and Light Gradient Boosting Machine (LightGBM) algorithms were trained to predict unmet HRSNs. Model performance was evaluated using 10-fold cross-validation and area under the receiver operating characteristic curve (AUROC). For overall HRSN prediction, LightGBM (AUROC, 64.5%, 95%CI: 64.3, 64.7) performed slightly better than logistic regression (61.4%), RF (63.7%), and XGBoost (60.3%). Similar performances were observed predicting individual HRSNs. Model performances were modest; however, they establish a benchmark for predictive performance achievable using only routinely available demographic data and provide a foundation for incorporating additional clinical and area-level social determinants of health data.

In this work, we demonstrate the unprecedented value of NIH's "All of Us Research Program" (AoURP) dataset in studying maternal morbidity and building predictive machine learning (ML) models across heterogeneous populations in the United States. We developed robust and data-driven preprocessing pipelines to curate a longitudinal, multi-site, multimodal, and demographically diverse pregnancy dataset (20,253 subjects; 27,525 pregnancy episodes) from AoURP data, using electronic health records (EHR) (Conditions, Labs, Measurements) and survey responses (Social Determinant of Health (SDoH)), focusing on 7 crucial maternal health adverse outcomes. After characterizing data quality, missingness, and heterogeneity, we performed statistical correlation analysis to identify risk factors. We subsequently developed XGBoost and sequential LSTM models to predict the adverse outcomes, reaching state-of-the-art performance for multiple outcomes. We conducted model interpretability post-hoc analysis to understand success points and fairness analysis to evaluate implications for socio-economic disparities. Four practicing physicians reviewed the set of statistically significant and ML model identified features to assess their clinical validity and novelty. Most features identified through either statistical correlations or ML feature importance analysis aligned with known clinical risk factors. Several features were identified that the ML models used but that are not currently used in clinical practice and may merit further clinical investigation. Fairness analysis revealed certain associations with SDoH and age highlight areas that warrant continued monitoring. Overall, we demonstrate that meaningful populational level patterns can be extracted, and high-performing machine learning models can be trained on this longitudinal, diverse, multi-site dataset. Important risk features, particularly novel ones identified, if validated, could inform new strategies for maternal care or enable development and validation of outcome-specific, clinically deployable ML models.

Objective. To introduce PsiBench, a clinically validated medication-safety benchmark for evaluating large language models (LLMs) against the standards used to certify hospital computerized provider order entry (CPOE) and electronic health record (EHR) systems, and a non-overlapping three-tier evaluation framework separating highest-stakes discrimination, the operational CDS regime, and category-correct alerting. Materials and Methods. PsiBench comprises 492 medication-safety scenarios across 11 safety categories, created by clinical pharmacology experts whose work underpins an annualized testing procedure used by more than 2,000 U.S. hospitals. The three-tier framework partitions the scenarios non-overlappingly: Discrimination (98 scenarios, 50 fatal vs 48 deception, near-balanced 51%/49%); Operational (394 scenarios, 261 serious unsafe plus 133 safe including 41 Excessive Alerts reclassified as operational negatives); and Attribution (311 alert-required scenarios). We evaluated 40 frontier LLMs from 10 providers over 3 runs per scenario at temperature 0.2 (or the provider default where temperature is not configurable), yielding 59,040 evaluations conducted April 21-23, 2026. Results. Headline binary performance on the full benchmark spans a wide range across the 40 models: F1 78.5%-92.3%, accuracy 65.4%-89.8%, sensitivity 81.4%-100.0%, specificity 6.1%-81.8%. Leading models by F1 (o4-mini 92.3%; o3 92.2%) pair high sensitivity with meaningful specificity; three models saturate sensitivity at 100% but fall below 25% specificity, indistinguishable from a naive always-alert classifier. The wide spread on a single headline metric motivates tier-specific analyses, developed in a separate clinical paper. Discussion and Conclusion. PsiBench and the three-tier framework operationalize a rigorous evaluation rubric for LLM medication safety, grounded in two decades of national hospital audit experience. The framework generalizes to any binary medication-safety classifier (rule-based, conventional ML, or LLM-driven), supporting tier-aware model selection and post-deployment surveillance.

Background Polyendocrine Metabolic Ovarian Syndrome (PMOS), formerly known as Polycystic Ovary Syndrome (PCOS), is a prevalent endocrine disorder with high rates of undiagnosed cases globally. Accessible screening tools are needed to facilitate appropriate management and earlier intervention. As PMOS is frequently characterised by oligo-anovulation, the absence of the characteristic rise in basal body temperature typically seen in ovulatory cycles may serve as a physiological marker for the condition. Objective This study aimed to assess the feasibility of using machine learning to identify individuals with PMOS from temperature data collected by a body-worn device. Methods We used data from 387 users of a vaginal temperature monitor (OvuSenseTM) who responded to a questionnaire. The sample was restricted to individuals with at least three cycles with sufficient temperature data and whose PMOS case/control status could be determined from questions about prior clinical consultation for infertility and conditions for which they take medications. We randomly sampled three menstrual cycles for each participant and derived a set of cycle-level and user-level temperature features. Cycle-level features included cycle length and measures describing the temperature rise indicative of ovulation (e.g. temperature rise, cycle day of temperature rise start). We also constructed a reference cycle representing the typical bi-phasic cycle pattern (created using cycles from those without known fertility conditions) and used this to derive features describing how much a participant's cycles differed from this reference. The cycle-level features were aggregated into user-level features by taking the minimum, maximum, median, and range of the cycle-level features across the three selected cycles for each participant. We used 5-fold nested cross-validation to evaluate the extent that PMOS could be predicted, at the cycle and user levels, using Logistic Regression (LR), Support Vector Machine (SVM), and Random Forest (RF). Results The average age of participants was 31.97 years (SD=4.58), with 49.6% having a self-reported PMOS diagnosis. The models demonstrated moderate discrimination, with cycle-level AUC-ROC scores ranging from 0.64 (SD=0.02) (LR) to 0.68 (SD=0.04) (RF), and user-level scores ranging from 0.65 (SD=0.07) (LR) to 0.70 (SD=0.04) (RF). All models were reasonably calibrated, though confidence intervals were wide (e.g. RF cycle-level: calibration slope = 0.83 (95% confidence interval [CI]: 0.68, 1.00), calibration intercept = 0.02 (95% CI: -0.11, 0.14); user-level: slope = 0.88 (95% CI: 0.69, 1.15), intercept = -0.01 (95% CI: -0.22, 0.16)). Conclusions This study demonstrates the potential of using body temperature from digital health devices to identify those with PMOS. Such a passive approach to identifying PMOS could help to identify undiagnosed PMOS in those who have not actively sought a diagnosis. Further research is needed to assess its predictive performance and acceptability in a general population using more widely used digital devices.

Diagnostic errors, including misdiagnoses and delayed clinical diagnoses, could affect outcomes of a significant patient population, particularly individuals presenting with rare diseases or non-specific symptoms. From rule-based diagnostic decision supporting systems (DDSS) to large language model (LLM) based tools for clinical reasoning have been developed to address these limitations. However, existing DDSS are often proprietary and difficult to integrate, and recent LLM-based tools remain hindered by operational challenges such as cost, resources constraint, and privacy concerns. Moreover, existing systems interpret electronic medical records (EMR) and generate diagnoses separately, limiting continuous evidence-based analysis and imposing repeated clinician involvement. In this paper, we present DDx-Finder, an open-source framework that leverages Model Context Protocol (MCP) servers for direct EMR and literature access, enabling prompt-driven clinical state extraction and reliable case-report re- trieval via generating searching query by LLM, while addressing limitations related to resource demands and privacy concerns. A clinical case study demonstrates the systems feasibility and its potential to provide accessible, transparent, and systematic differential diagnostic support for complex cases.

Clinical informatics pipelines increasingly compute validated clinical endpoints from upstream NLP outputs. Even when the endpoint is defined by an established rubric, translating that rubric across representations - natural language instructions, program logic, and reference implementations - can introduce specification drift, where ostensibly equivalent calculators yield meaningfully different scores. We study this phenomenon for the Expanded Disability Status Scale (EDSS), a standard measure of disability in multiple sclerosis. Holding constant a shared set of functional system (FS) subscores extracted by a large language model (LLM), we compare EDSS values computed across three representations of the same scoring rubric: prompt-executed natural language, LLM-generated code, and a canonical reference implementation. We characterize disagreement structure, distributional shifts, and clinically salient boundary flips, and we propose an audit workflow that treats endpoint computation as a first-class verification target in clinical NLP systems.